i am trying to write to azure blob using the entra ID it gives the following error : ERROR AzureNativeFileSystemStore: Encountered Storage Exception for delete on Blob: https://<acc>.blob.core.windows.net/<container>/_$azuretmpfolder$/ba8b5467-b477-460a-97c8-662223b4d21epart-0082fb-41b238001672.c000.snappy.parquet , Exception Details: This request is not authorized to perform this operation using this permission. Error Code: AuthorizationPermissionMismatch Please help . files are being written in _temprory folder

Hi Prem - Thanks for reaching out. Based on the exception, the SPN seem to be missing some permission in order to perform the write operation. Are you making of RBAC or ACLs? If ACL's, you can try using RBAC with role such as Storage blob Data Contributor that shall help in the isolation. Another pointer to check is if there are any networking level restriction on the account level? Please let us know if you have any further queries. I’m happy to assist you further. Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Okay I have changed that: It still shows the same error where its not able to delete _$azuretmpfolder$ .Can you please help on that?

Saprk Azure blob Writer with Principal Secret (entraID)

Accepted answer

Amrinder Singh 3,260 Reputation points Microsoft Employee

2024-05-15T02:28:14.22+00:00

Hi Prem - Thanks for reaching out.

Based on the exception, the SPN seem to be missing some permission in order to perform the write operation.

Are you making of RBAC or ACLs? If ACL's, you can try using RBAC with role such as Storage blob Data Contributor that shall help in the isolation.

Another pointer to check is if there are any networking level restriction on the account level?

Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

1 person found this answer helpful.
Prem 25 Reputation points

2024-05-15T06:51:26.2766667+00:00

This is already done. The App has all the access, networking is: allowed from all networks.

Prem 25 Reputation points

2024-05-15T06:58:24.1+00:00

Please find the network screenshot below.

Amrinder Singh 3,260 Reputation points Microsoft Employee

2024-05-15T08:11:36.93+00:00

Hi Prem - Thanks for sharing the information.

You are making use of Storage Account Contributor which is more of management plane role and doesn't have any data plane permissions.

https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-account-contributor

Kindly try using "Storage Blob Data Contributor" role and test it ahead.

Amrinder Singh 3,260 Reputation points Microsoft Employee

2024-05-15T10:48:14.62+00:00

Hi Prem - Thanks for testing that.

Can you share the complete exception stack trace along with failed request ID if any?

Prem 25 Reputation points

2024-05-15T11:04:50.5933333+00:00

errorLog.txt

Please check the error logs file provided above.

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Amrinder Singh 3,260 Reputation points Microsoft Employee

2024-05-15T15:09:40.78+00:00

Hi Prem -

If the role of storage blob data contributor is there and the access is allowed from all network then it should allow you to perform the deletion.

I tried looking at the logs you shared however couldn't find anything much and few details are removed for compliance reason so nothing conclusive from there.

By any chance you have any kind of PE configuration in place (2nd tab in the snippet)?

If yes, you can try removing that once and test it ahead. If it doesn't work or you don't have the PE configurations at all, then I would recommend raising a support ticket to take a look at the backend logs for more deeper analysis.

Hope that helps!

Prem 25 Reputation points

2024-05-15T15:14:55.44+00:00

Thanks, I have created a new container now and used the same by giving the defined permissions. It's working now. Thanks for your help. Just one more thing is there a documented spark code (for best proctice ) to connect to azure blob (read/write) using principal secret? Please share the link.

Amrinder Singh 3,260 Reputation points Microsoft Employee

2024-05-16T16:08:19.8066667+00:00

Thanks Prem for the update. Glad to hear that it worked but that appeared to be some weird behavior. If you have deleted and created new container, even support might not be able to backtrack much. But in case that container still exists, you can reach out to support for further troubleshooting ahead.

If you are referring to Databricks spark, below is the ref link for the same:

https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-use-databricks-spark

In case you are referring to open-source spark configuration, you can explore multiple 3rd party links that you can refer but more like syntax would remain the same.

For in general best practices regarding working with ADLS Gen2 account, you can refer to the below link:

https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices

Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Sign in to comment

Share via

Saprk Azure blob Writer with Principal Secret (entraID)

1 additional answer